Explore City venues (Chennai)¶
The city venue data will be use to explore the cities.
- Firstly the most no a venue will be extracted from a particular city.
- Then the top 10 most common venue for each neighborhood will be extracted and then the neighborhood will be clustered by kmeans on the basis of most common venues
In [319]:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
sns.set()
Initialize an array of cities in which we are interested in¶
In [320]:
# initialize cities array and perform exploratory data analysis on the selected city
cities = ['Delhi','Mumbai','Kolkata','Chennai']
city = cities[3]
city_venues = pd.read_csv(city + '_venues.csv',index_col = 0)
city_venues.head()
Out[320]:
Exploratory data Analysis¶
In [321]:
# see number of venues per neighbourhood
city_venues.groupby('Neighborhood').count().head()
Out[321]:
One hotting the dataframe¶
In [322]:
# one hot encoding
city_onehot = pd.get_dummies(city_venues[['Venue Category']], prefix="", prefix_sep="")
# add neighborhood column back to dataframe
city_onehot['Neighborhood'] = city_venues['Neighborhood']
# move neighborhood column to the first column
fixed_columns = [city_onehot.columns[-1]] + list(city_onehot.columns[:-1])
city_onehot = city_onehot[fixed_columns]
city_onehot['City'] = city
city_onehot.head()
Out[322]:
In a city finding the total no. of venues of a particular type and arranginf the dataframe¶
In [323]:
# group the data per neighborhood
city_grouped = city_onehot.groupby('City').sum().reset_index()
city_grouped = city_grouped.transpose()
city_grouped.columns = city_grouped.iloc[0]
city_grouped.drop(city_grouped.index[[0]],inplace=True)
city_grouped.head()
Out[323]:
Sorting the data frame on the basis of frequency, taking the top 10 and plotting them.¶
In [324]:
city_grouped.sort_values([city],ascending=False,inplace=True)
city_grouped.iloc[0:10].plot(kind='bar',figsize=(10,5))
plt.title(city + ' Venue distribution')
plt.ylabel('Total no of venues in the City')
plt.xlabel('Venue Categories')
Out[324]:
Now we will find the top 10 most common venues¶
In [325]:
# First, let's write a function to sort the venues in descending order
def return_most_common_venues(row, num_top_venues):
row_categories = row.iloc[1:]
row_categories_sorted = row_categories.sort_values(ascending=False)
return row_categories_sorted.index.values[0:num_top_venues]
In [326]:
city_grouped = city_onehot.groupby('Neighborhood').mean().reset_index()
city_grouped.head()
Out[326]:
In [327]:
# Now let's create the new dataframe and display the top 10 venues for each neighborhood.
num_top_venues = 10
indicators = ['st', 'nd', 'rd']
# create columns according to number of top venues
columns = ['Neighborhood']
for ind in np.arange(num_top_venues):
try:
columns.append('{}{} Most Common Venue'.format(ind+1, indicators[ind]))
except:
columns.append('{}th Most Common Venue'.format(ind+1))
# create a new dataframe
neighborhoods_venues_sorted = pd.DataFrame(columns=columns)
neighborhoods_venues_sorted['Neighborhood'] = city_grouped['Neighborhood']
for ind in np.arange(city_grouped.shape[0]):
neighborhoods_venues_sorted.iloc[ind, 1:] = return_most_common_venues(city_grouped.iloc[ind, :], num_top_venues)
neighborhoods_venues_sorted.head()
Out[327]:
After finding the top 10 venues in a neighborhood, the neighborhood will be clustered by kmeans algorithm¶
In [328]:
# import necessary packages
from sklearn.datasets import load_iris
from sklearn.cluster import KMeans
import matplotlib.pyplot as plt
Find optimal k for the clustering process.¶
In [329]:
city_grouped_clustering = city_grouped.drop('Neighborhood', 1)
sse = {}
for k in range(1, 10):
# run k-means clustering
kmeans = KMeans(n_clusters=k, random_state=0,max_iter = 1000).fit(city_grouped_clustering)
city_grouped_clustering["clusters"] = kmeans.labels_
sse[k] = kmeans.inertia_ # Inertia: Sum of distances of samples to their closest cluster center
plt.figure()
plt.plot(list(sse.keys()), list(sse.values()))
plt.xlabel("Number of cluster")
plt.ylabel("SSE")
plt.show()
From the graph we find the elbow point an then use it to cluster the neighborhoods.¶
In [330]:
# set number of clusters
kclusters = 6
city_grouped_clustering = city_grouped.drop('Neighborhood', 1)
# run k-means clustering
kmeans = KMeans(n_clusters=kclusters, random_state=0).fit(city_grouped_clustering)
# check cluster labels generated for each row in the dataframe
kmeans.labels_[0:10]
Out[330]:
In [331]:
df = pd.read_csv(city + '_subdiv.csv',index_col=0)
df.head()
Out[331]:
Add clustering labels to the dataframe¶
In [332]:
# add clustering labels
neighborhoods_venues_sorted.insert(0, 'Cluster Labels', kmeans.labels_)
city_merged = df
# merge toronto_grouped with toronto_data to add latitude/longitude for each neighborhood
city_merged = city_merged.join(neighborhoods_venues_sorted.set_index('Neighborhood'), on='Neighborhood')
# drop the column with nan values after join
city_merged.dropna(inplace=True)
city_merged.head() # check the last columns!
Out[332]:
After clustering the data will be visualized in folium.¶
In [333]:
# Matplotlib and associated plotting modules
import matplotlib.cm as cm
import matplotlib.colors as colors
import folium
Create a function to ge the lat and long of the center of city using geocoder API¶
In [334]:
# use geocoder library, if not present use !conda install -c conda-forge geocoder
import geocoder
# Google API key is required for the geocoder library to work, save the API key in OS environment variables as GOOGLE_API_KEY
# and then access thay key here
import os
# Use BING_API_KEY when choosing to use bing geocoding instead of google geocoding.
BING_API_KEY = 'AksNN-3luSfNBssyZ3Ju4i78nIrFLt1UtYo--YWQj9oyfxSwyXkdsqykWk3FeTXB' # os.environ['BING_API_KEY']
# This function will take an adress and return the latlng of that adress
def get_latlng(address):
# using bing geocoder API since it is better.
g = geocoder.bing(address, key = BING_API_KEY)
return pd.Series(g.latlng)
In [335]:
# get latitude and longitude of city to center the map
latitude, longitude = get_latlng(city)
print('Lat : ',latitude,' Long : ',longitude)
Visualize the neighborhoods before clustering¶
In [336]:
# Function takes in a data frame with Latitude, Longitude, Neighborhood and City columns and shows it on map
def visualize_area_in_map(data):
# add markers to map
for lat, lng, neighborhood, city in zip(data['Latitude'], data['Longitude'], data['Neighborhood'], data['City']):
label = '{}, {}'.format(neighborhood, city)
label = folium.Popup(label, parse_html=True)
folium.CircleMarker(
[lat, lng],
radius=2,
popup=label,
color='blue',
fill=True,
fill_color='#3186cc',
fill_opacity=0.7,
parse_html=False).add_to(map)
return map
In [337]:
# create map of Toronto using latitude and longitude values
map = folium.Map(location=[latitude, longitude], zoom_start=10)
# data to be used for map
data = df.dropna()
visualize_area_in_map(data)
Out[337]:
Visualize the clusters¶
In [338]:
# create map
map_clusters = folium.Map(location=[latitude, longitude], zoom_start=10)
# set color scheme for the clusters
x = np.arange(kclusters)
ys = [i + x + (i*x)**2 for i in range(kclusters)]
colors_array = cm.rainbow(np.linspace(0, 1, len(ys)))
rainbow = [colors.rgb2hex(i) for i in colors_array]
# add markers to the map
markers_colors = []
for lat, lon, poi, cluster in zip(city_merged['Latitude'], city_merged['Longitude'], city_merged['Neighborhood'], city_merged['Cluster Labels']):
label = folium.Popup(str(poi) + ' Cluster ' + str(cluster), parse_html=True)
folium.CircleMarker(
[lat, lon],
radius=4,
popup=label,
color=rainbow[int(cluster)-1],
fill=True,
fill_color=rainbow[int(cluster)-1],
fill_opacity=0.9).add_to(map_clusters)
map_clusters
Out[338]:
Exploring the clusters¶
In [339]:
# print the cluster
city_merged.loc[city_merged['Cluster Labels'] == 0, city_merged.columns[[1] + list(range(5, city_merged.shape[1]))]]
Out[339]:
In [340]:
city_merged.loc[city_merged['Cluster Labels'] == 1, city_merged.columns[[1] + list(range(5, city_merged.shape[1]))]]
Out[340]:
In [341]:
city_merged.loc[city_merged['Cluster Labels'] == 2, city_merged.columns[[1] + list(range(5, city_merged.shape[1]))]]
Out[341]:
In [342]:
city_merged.loc[city_merged['Cluster Labels'] == 3, city_merged.columns[[1] + list(range(5, city_merged.shape[1]))]]
Out[342]:
In [343]:
city_merged.loc[city_merged['Cluster Labels'] == 4, city_merged.columns[[1] + list(range(5, city_merged.shape[1]))]]
Out[343]:
In [344]:
city_merged.loc[city_merged['Cluster Labels'] == 5, city_merged.columns[[1] + list(range(5, city_merged.shape[1]))]]
Out[344]:
In [ ]: